Detecting Differential Item Functioning in Multidimensional Graded Response Models With Recursive Partitioning.

Differential item functioning (DIF) is a common challenge when examining latent traits in large scale surveys. In recent work, methods from the field of machine learning such as model-based recursive partitioning have been proposed to identify subgroups with DIF when little theoretical guidance and many potential subgroups are available. On this basis, we propose and compare recursive partitioning techniques for detecting DIF with a focus on measurement models with multiple latent variables and ordinal response data. We implement tree-based approaches for identifying subgroups that contribute to DIF in multidimensional latent variable modeling and propose a robust, yet scalable extension, inspired by random forests. The proposed techniques are applied and compared with simulations. We show that the proposed methods are able to efficiently detect DIF and allow to extract decision rules that lead to subgroups with well fitting models.

Authenticity Scale: Validity and Reliability Evidence in a Sample from Brazil and Portugal / Escala de autenticidad: evidencia de validez y confiabilidad en una muestra de Brasil y Portugal

Este estudio presenta distintos tipos de evidencias de validez y confiabilidad de la Escala de Autenticidad (AS) en una muestra de Brasil y Portugal. El estudioconsiste en una encuesta con 1.077 brasileños y 622 portugueses. Se testó el modelo con tres factores correlacionados (autoalienación, vivir auténtico y aceptación de la influencia externa), el modelo unidimensional y el modelo bifactorial. Se retuvo el modelo con tres factores correlacionados, con las tres subescalas alcanzando confiabilidad moderada a buena. Análisis factorial confirmatorio multigrupo sugirió invariancia escalar para cultura, género, edad, educación, ocupación y preocupación e impacto relacionados con Covid. Los ítems fueron evaluados por graded response model(GRM), sugiriendo que las tres subescalas no discriminan las personas con altos rasgos de autenticidad. GRM y estadísticas descriptivas indican que la escala de puntuación es inapropiada, particularmente para la subescala vivir auténtico, que es afectada por efecto techo. Las asociaciones con presencia de sentido mostraron evidencia adicional de validez. A pesar de las limitaciones, la AS es una medida adecuada para evaluar la autenticidad en diferentes grupos. Se discuten posibles modificaciones para el aprimoramiento de la AS.(AU)

This study introduces distinct types of validity and reliability evidence of the Authenticity Scale (AS) in a sample from Brazil and Portugal. It consists of an online survey with 1,077 Brazilian citizens and 622 Portuguese citizens. The study tested the model with three correlated factors (self-alienation, authentic living, and accepting external influence), the unidimensional model, and the bifactor model. The model with three correlated factors was retained, with the three subscales demonstrating moderate to good reliability. Multigroup confirmatory factor analysis suggested scalar invariance across culture, gender, age, education, occupation, and Covid-related concern and impact. The items were assessed by graded response model (GRM), which suggested that the three subscales are not able to distinguish respondents with high authenticity traits. GRM and descriptive statistics indicated that the rating-scale is inappropriate, particularly for authentic living subscale, which is affected by ceiling effect. Associations with presence of meaning showed additional validity evidence. Despite the limitations, the AS is an effective measure to assess authenticity across different groups. Potential modifications for the improvement of the AS are discussed.(AU)

Are the Steps on Likert Scales Equidistant? Responses on Visual Analog Scales Allow Estimating Their Distances.

A recurring question regarding Likert items is whether the discrete steps that this response format allows represent constant increments along the underlying continuum. This question appears unsolvable because Likert responses carry no direct information to this effect. Yet, any item administered in Likert format can identically be administered with a continuous response format such as a visual analog scale (VAS) in which respondents mark a position along a continuous line. Then, the operating characteristics of the item would manifest under both VAS and Likert formats, although perhaps differently as captured by the continuous response model (CRM) and the graded response model (GRM) in item response theory. This article shows that CRM and GRM item parameters hold a formal relation that is mediated by the form in which the continuous dimension is partitioned into intervals to render the discrete Likert responses. Then, CRM and GRM characterizations of the items in a test administered with VAS and Likert formats allow estimating the boundaries of the partition that renders Likert responses for each item and, thus, the distance between consecutive steps. The validity of this approach is first documented via simulation studies. Subsequently, the same approach is used on public data from three personality scales with 12, eight, and six items, respectively. The results indicate the expected correspondence between VAS and Likert responses and reveal unequal distances between successive pairs of Likert steps that also vary greatly across items. Implications for the scoring of Likert items are discussed.

Analyzing the psychometric properties of the PHQ-9 using item response theory in a Chinese adolescent population.

BACKGROUND: People are more likely to fall victim to depression during adolescence since it is a period of rapid biopsychosocial transformation. Despite this, most depression research has concentrated on clinical issues, and evaluating depressive symptoms in teenagers is not as widespread. This study used item response theory (IRT) to examine the psychometric properties of the Patient Health Report scale (PHQ-9) in Chinese adolescents. Meanwhile, item function difference tests were used to check whether there were differences in depression symptoms in this group based on education and gender. METHODS: In this research, the PHQ-9 was employed as a measurement tool, and 5958 valid data points were obtained from 12 secondary schools in China (Mage = 13.484; SDage = 1.627; range 11-19 years; 52.17% boys). RESULTS: IRT shows that all items of the PHQ-9 satisfy monotonicity, unidimensionality and local independence and that they have good psychometric properties. Furthermore, DIF analysis revealed gender and educational disparities in adolescent depressive symptoms. CONCLUSION: The study indicates that the PHQ-9 possesses favourable psychometric properties for use in Chinese adolescents. As a result, it serves as a valuable tool for effectively screening depressive symptoms in adolescents. It provides a foundation for prioritizing the development of secondary school students' physical and mental health.

A Gibbs-INLA algorithm for multidimensional graded response model analysis.

In this paper, we propose a novel Gibbs-INLA algorithm for the Bayesian inference of graded response models with ordinal response based on multidimensional item response theory. With the combination of the Gibbs sampling and the integrated nested Laplace approximation (INLA), the new framework avoids the cumbersome tuning which is inevitable in classical Markov chain Monte Carlo (MCMC) algorithm, and has low computing memory, high computational efficiency with much fewer iterations, and still achieve higher estimation accuracy. Therefore, it has the ability to handle large amount of multidimensional response data with different item responses. Simulation studies are conducted to compare with the Metroplis-Hastings Robbins-Monro (MH-RM) algorithm and an application to the study of the IPIP-NEO personality inventory data is given to assess the performance of the new algorithm. Extensions of the proposed algorithm for application on more complicated models and different data types are also discussed.

Psychometric properties of the Chinese version of the PROMIS-Cancer-Anxiety item bank assessed using a graded response model.

Objective: This study aimed to examine the psychometric properties of the Chinese version of the Patient-Reported Outcome Measurement Information System (PROMIS)-Cancer-Anxiety item bank using a graded response model in a sample of patients with cancer. Methods: A cross-sectional study was conducted and the Chinese version of the PROMIS-Cancer-Anxiety item bank was used to measure anxiety in patients with cancer. The unidimensional structure of the item bank was evaluated using principal component analysis. Residual correlations and the graphs of item mean scores conditional on the rest scores were examined to evaluate the local independence and monotonicity of the items, respectively. Item characteristics were described using item parameter estimates and item information. Operating characteristic curves (OCCs) and test information curve (TIC) were also plotted. Measurement invariance across age, gender, and education level was assessed to identify possible differential item functioning (DIF). Results: A total of 1075 patients with cancer were enrolled. Under the assumptions of unidimensionality, local independence, and monotonicity, the discrimination parameters a ranged from 2.30 to 5.47, and the threshold parameters b ranged from b1 = -2.87 to b4 = 3.21 with proper intervals. Completely overlapped category curves were not observed among the OCCs of any items. Item information and TIC showed that the item bank had a wide measurement range. The DIFs for age, gender, and education level for all items were not remarkable. Conclusions: The results supported using the Chinese version of the PROMIS-Cancer-Anxiety item bank to measure anxiety and develop a computerized adaptive testing (CAT) system for anxiety in patients with cancer.

Using Auxiliary Item Information in the Item Parameter Estimation of a Graded Response Model for a Small to Medium Sample Size: Empirical Versus Hierarchical Bayes Estimation.

Marginal maximum likelihood estimation (MMLE) is commonly used for item response theory item parameter estimation. However, sufficiently large sample sizes are not always possible when studying rare populations. In this paper, empirical Bayes and hierarchical Bayes are presented as alternatives to MMLE in small sample sizes, using auxiliary item information to estimate the item parameters of a graded response model with higher accuracy. Empirical Bayes and hierarchical Bayes methods are compared with MMLE to determine under what conditions these Bayes methods can outperform MMLE, and to determine if hierarchical Bayes can act as an acceptable alternative to MMLE in conditions where MMLE is unable to converge. In addition, empirical Bayes and hierarchical Bayes methods are compared to show how hierarchical Bayes can result in estimates of posterior variance with greater accuracy than empirical Bayes by acknowledging the uncertainty of item parameter estimates. The proposed methods were evaluated via a simulation study. Simulation results showed that hierarchical Bayes methods can be acceptable alternatives to MMLE under various testing conditions, and we provide a guideline to indicate which methods would be recommended in different research situations. R functions are provided to implement these proposed methods.

Latent trait models for perceived risk assessment using a Covid-19 data survey.

Aim of the contribution is analyzing potential events that may negatively impact individuals, assets, and/or the environment, and making judgments about the perceived personal and social riskiness of Covid-19 compared to other hazards belonging to health (AIDS, cancer, infarction), environmental (climate change), behavioral (serious car accidents), and technological (nuclear weapons) domains. The comparative risk analysis has been performed on a survey data collected during the first Italian Covid-19 lockdown. An item response theory model for polytomously scored items has been implemented for the analysis of the positioning of Covid-19 with respect to the other hazards in terms of perceived risk. Among the attributes determining the hazard's perceived risk, Covid-19 distinguishes for the knowledge of risks from the hazard, media attention, and fear caused by the hazard in the peers. Besides, through a latent regression analysis, the role of some individual characteristics on the perceived risk for Covid-19 has been examined. Our contribution allows us to disentangle among several aspects of hazards and describe the main factors affecting the perceived risk. It also contributes to determine if existing control measures are perceived as adequate and the interest for new media with related impact on a person's reaction.

Graded Response Model Analysis and Computer Adaptive Test Simulation of the Depression Anxiety Stress Scale 21: Evaluation and Validation Study.

BACKGROUND: The Depression Anxiety Stress Scale 21 (DASS-21) is a mental health screening tool with conflicting studies regarding its factor structure. No studies have yet attempted to develop a computer adaptive test (CAT) version of it. OBJECTIVE: This study calibrated items for, and simulated, a DASS-21 CAT using a nonclinical sample. METHODS: An evaluation sample (n=580) was used to evaluate the DASS-21 scales via confirmatory factor analysis, Mokken analysis, and graded response modeling. A CAT was simulated with a validation sample (n=248) and a simulated sample (n=10,000) to confirm the generalizability of the model developed. RESULTS: A bifactor model, also known as the "quadripartite" model (1 general factor with 3 specific factors) in the context of the DASS-21, displayed good fit. All scales displayed acceptable fit with the graded response model. Simulation of 3 unidimensional (depression, anxiety, and stress) CATs resulted in an average 17% to 48% reduction in items administered when a reliability of 0.80 was acceptable. CONCLUSIONS: This study clarifies previous conflicting findings regarding the DASS-21 factor structure and suggests that the quadripartite model for the DASS-21 items fits best. Item response theory modeling suggests that the items measure their respective constructs best between 0θ and 3θ (mild to moderate severity).

Evaluating the psychometric properties of the fatigue severity scale using item response theory.

BACKGROUND: Fatigue is a common daily experience and a symptom of various disorders. While scholars have discussed the use of the Fatigue Severity Scale (FSS) using item response theory (IRT), the characteristics of the Japanese version are not yet examined. This study evaluated the psychometric properties of the FSS using IRT and assessed its reliability and concurrent validity with a general sample in Japan. METHODS AND MEASURES: A total of 1,007 Japanese individuals participated in an online survey, with 692 of them providing valid data. Of these, 125 participants partook in a re-test after approximately 18 days and had their longitudinal data analyzed. In addition, the graded response model (GRM) was used to assess the FSS items' characteristics. RESULTS: The GRM's results recommended using seven items and a 6-point scale. The FSS's reliability was acceptable. Furthermore, the validity was adequate from the results of correlation and regression analyses. The synchronous effects models demonstrated that the Multidimensional Fatigue Inventory (MFI) enhanced depression, and depression enhanced FSS. CONCLUSION: This study suggested that the Japanese version of the FSS should be a 7-item scale with a 6-point response scale. Further investigations may reveal the different aspects of fatigue assessed by the analyzed fatigue measures.

Longitudinal validation of the Fear of COVID-19 Scale in a nationwide United States sample: An item response theory model across three inflection points of the pandemic.

The COVID-19 pandemic's global emergence/spread caused widespread fear. Measurement/tracking of COVID-19 fear could facilitate remediation. Despite the Fear of COVID-19 Scale (FCV-19S)'s validation in multiple languages/countries, nationwide United States (U.S.) studies are scarce. Cross-sectional classical test theory-based validation studies predominate. Our longitudinal study sampled respondents to a 3-wave, nationwide, online survey. We calibrated the FCV-19S using a unidimensional graded response model. Item/scale monotonicity, discrimination, informativeness, goodness-of-fit, criterion validity, internal consistency, and test-retest reliability were assessed. Items 7, 6, and 3 consistently displayed very high discrimination. Other items had moderate-to-high discrimination. Items 3, 6, and 7 were most (items 1 and 5 the least) informative. [Correction added on 18 May 2023, after first online publication: In the preceding sentence, the term 'items one-fifth least' has been changed to 'items 1 and 5 the least'.] Item scalability was 0.62-0.69; full-scale scalability 0.65-0.67. Ordinal reliability coefficient was 0.94; test-retest intraclass correlation coefficient 0.84. Positive correlations with posttraumatic stress/anxiety/depression, and negative correlations with emotional stability/resilience supported convergent/divergent validity. The FCV-19S validly/reliably captures temporal variation in COVID-19 fear across the U.S.

Evaluation of Polytomous Item Locations in Multicomponent Measuring Instruments: A Note on a Latent Variable Modeling Procedure.

This note is concerned with evaluation of location parameters for polytomous items in multiple-component measuring instruments. A point and interval estimation procedure for these parameters is outlined that is developed within the framework of latent variable modeling. The method permits educational, behavioral, biomedical, and marketing researchers to quantify important aspects of the functioning of items with ordered multiple response options, which follow the popular graded response model. The procedure is routinely and readily applicable in empirical studies using widely circulated software and is illustrated with empirical data.

Farmers' response to maize production risks in Cameroon: An application of the criticality risk matrix model.

Farmers' inability to successfully participate in the maize value chain due to the threats of some risk factors is one of the key reasons for rising food insecurity. This study analyses farmers' responses to risks involved in maize production in Cameroon. Data on maize production risks were collected from smallholder maize farmers operating in selected communities along the River Sanaga. The Criticality Risk Matrix model was employed to determine the severity of these risks based on their criticality and likelihood of occurrence. After deriving farmers' risk preferences from the categorization of their farm decisions, a Multinomial Logit Regression model was then used to investigate the extent to which risk severity influences farm decisions. Additionally, a Graded Response Model was utilized to predict farmers' risk responses by categorizing their probable patterns of action. The results revealed that production risks with fatal consequences such as pest invasion had a negative significant influence on-farm decisions and their perceived threats were more likely to provoke risk-averse responses. Less fatal risk threats such as fertilizer unavailability, poor farm infrastructure, labour shortage, and health risks were all significant and induced risk-averse responses from farmers. Furthermore, gender, experience and employment status also significantly influence farm decision-making. The farmers' response patterns presented on the Graded Response Model Characteristic Curves further indicated the willingness of farmers to continue farm activities despite perceived risks and the probability of engaging in diversification as an added measure to mitigate risks. We recommended that more effective measures for information dissemination on production risks as well as sustained Extension Service support to mitigate these risks be made available to farmers.

Measurement invariance of the Liebowitz Social Anxiety Scale-Self-Report.

OBJECTIVE: The Liebowitz Social Anxiety Scale-Self Report (LSAS-SR) is a self-report measure of social anxiety (SA), which has shown adequate psychometric properties across cultures. However, no study has systematically evaluated its measurement invariance (MI) between (a) individuals with and without a diagnosis of social anxiety disorder (SAD) and (b) males and females. The current study addresses this issue. METHODS: We collected data on 257 (158 females) Italian individuals diagnosed with SAD and 356 (232 females) community-dwelling adults. RESULTS: We initially found support for the unidimensionality of the Italian LSAS-SR measurement model in all samples. Using the Graded Response Model, we obtained evidence of partial MI and differential item functioning between community-dwelling and SAD-diagnosed individuals and evidence of strong MI between male and female participants. CONCLUSIONS: The results of this study suggest that the Italian LSAS-SR measures the same trait in the same way across the symptom continuum and sexes, making it a psychometrically sound tool for assessment, screening, and research purposes.

Overfactoring in rating scale data: A comparison between factor analysis and item response theory.

Educational and psychological measurement is typically based on dichotomous variables or rating scales comprising a few ordered categories. When the mean of the observed responses approaches the upper or the lower bound of the scale, the distribution of the data becomes skewed and, if a categorical factor model holds in the population, the Pearson correlation between variables is attenuated. The consequence of this correlation attenuation is that the traditional linear factor model renders an excessive number of factors. This article presents the results of a simulation study investigating the problem of overfactoring and some solutions. We compare five widely known approaches: (1) The maximum-likelihood factor analysis (FA) model for normal data, (2) the categorical factor analysis (FAC) model based on polychoric correlations and maximum likelihood (ML) estimation, (3) the FAC model estimated using a weighted least squares algorithm, (4) the mean corrected chi-square statistic by Satorra-Bentler to handle the lack of normality, and (5) the Samejima's graded response model (GRM) from item response theory (IRT). Likelihood-ratio chi-square, parallel analysis (PA), and categorical parallel analysis (CPA) are used as goodness-of-fit criteria to estimate the number of factors in the simulation study. Our results indicate that the maximum-likelihood estimation led to overfactoring in the presence of skewed variables both for the linear and categorical factor model. The Satorra-Bentler and GRM constitute the most reliable alternatives to estimate the number of factors.

An Empirical Identification Issue of the Bifactor Item Response Theory Model.

Using the bifactor item response theory model to analyze data arising from educational and psychological studies has gained popularity over the years. Unfortunately, using this model in practice comes with challenges. One such challenge is an empirical identification issue that is seldom discussed in the literature, and its impact on the estimates of the bifactor model's parameters has not been demonstrated. This issue occurs when an item's discriminations on the general and specific dimensions are approximately equal (i.e., the within-item discriminations are similar in strength), leading to difficulties in obtaining unique estimates for those discriminations. We conducted three simulation studies to demonstrate that within-item discriminations being similar in strength creates problems in estimation stability. The results suggest that a large sample could alleviate but not resolve the problems, at least when considering sample sizes up to 4,000. When the discriminations within items were made clearly different, the estimates of these discriminations were more consistent across the data replicates than that observed when the discriminations within the items were similar. The results also show that the similarity of an item's discriminatory magnitudes on different dimensions has direct implications on the sample size needed in order to consistently obtain accurate parameter estimates. Although our goal was to provide evidence of the empirical identification issue, the study further reveals that the extent of similarity of within-item discriminations, the magnitude of discriminations, and how well the items are targeted to the respondents also play factors in the estimation of the bifactor model's parameters.

Comparison of unweighted and item response theory-based weighted sum scoring for the Nine-Questions Depression-Rating Scale in the Northern Thai Dialect.

BACKGROUND: The Nine-Questions Depression-Rating Scale (9Q) has been developed as an alternative assessment tool for assessing the severity of depressive symptoms in Thai adults. The traditional unweighted sum scoring approach does not account for differences in the loadings of the items on the actual severity. Therefore, we developed an Item Response Theory (IRT)-based weighted sum scoring approach to provide a scoring method that is more precise than the unweighted sum score. METHODS: Secondary data from a study on the criterion-related validity of the 9Q in the northern Thai dialect was used in this study. All participants were interviewed to obtain demographic data and screened/evaluated for major depressive disorder and the severity of the associated depressive symptoms, followed by diagnosis by a psychiatrist for major depressive disorder. IRT models were used to estimate the discrimination and threshold parameters. Differential item functioning (DIF) of responses to each item between males and females was compared using likelihood-ratio tests. The IRT-based weighed sum scores of the individual items are defined as the linear combination of individual response weighted with the discrimination and threshold parameters divided by the plausible maximum score based on the graded-response model (GRM) for the 9Q score (9Q-GRM) or the nominal-response model (NRM) for categorical combinations of the intensity and frequency of symptoms from the 9Q responses (9QSF-NRM). The performances of the two scoring procedures were compared using relative precision. RESULTS: Of the 1,355 participants, 1,000 and 355 participants were randomly selected for the developmental and validation group for the IRT-based weighted scoring, respectively. the gender-related DIF were presented for items 2 and 5 for the 9Q-GRM, while most items (except for items 3 and 6) for the 9QSF-NRM, which could be used to separately estimate the parameters between genders. The 9Q-GRM model accounting for DIF had a higher precision (16.7%) than the unweighted sum-score approach. DISCUSSION: Our findings suggest that weighted sum scoring with the IRT parameters can improve the scoring when using 9Q to measure the severity of the depressive symptoms in Thai adults. Accounting for DIF between the genders resulted in higher precision for IRT-based weighted scoring.

LASSO-Based Pattern Recognition for Replenished Items With Graded Responses in Multidimensional Computerized Adaptive Testing.

As a branch of statistical latent variable modeling, multidimensional item response theory (MIRT) plays an important role in psychometrics. Multidimensional graded response model (MGRM) is a key model for the development of multidimensional computerized adaptive testing (MCAT) with graded-response data and multiple traits. This paper explores how to automatically identify the item-trait patterns of replenished items based on the MGRM in MCAT. The problem is solved by developing an exploratory pattern recognition method for graded-response items based on the least absolute shrinkage and selection operator (LASSO), which is named LPRM-GR and facilitates the subsequent parameter estimation of replenished items and helps maintaining the effectiveness of item replenishment in MCAT. In conjunction with the proposed approach, the regular BIC and weighted BIC are applied, respectively, to select the optimal item-trait patterns. Simulation for evaluating the LPRM-GR in pattern recognition accuracy of replenished items and the corresponding item estimation accuracy is conducted under multiple conditions across different numbers with respect to dimensionality, response-category numbers, latent trait correlation, stopping rules, and item selection criteria. Results show that the proposed method with the two types of BIC both have good performance in pattern recognition for item replenishment in the two- to four-dimensional MCAT with the MGRM, for which the weighted BIC is generally superior to the regular BIC. The proposed method has relatively high accuracy and efficiency in identifying the patterns of graded-response items, and has the advantages of easy implementation and practical feasibility.

Re-assessing the Psychometric Properties of Stress Appraisal Measure in Ghana Using Multidimensional Graded Response Model.

Despite the widespread use of the stress appraisal measure questionnaire in sport psychology literature, information on the psychometric properties of this survey instrument across different cultures and samples is still lacking. This study sought to validate the stress appraisal measure among male football players in the Ghana's Premier League using the multidimensional item response theory. The descriptive cross-sectional survey design was adopted to recruit 424 footballers from the 2020/2021 Ghana Premier League season using the census approach. The 28-item Stress Appraisal Measure was used to assess six (6) appraisal mechanisms under primary and secondary cognitive appraisals. The ordered polytomous item response theory was used for analyzing the data. The study found that although some items were problematic, the majority of them were found to have good item parameters, effective scale option functioning, and provided adequate empirical information in the measurement of stress appraisal. This research concluded that the stress appraisal measure has promising applicability among male footballers who participated in the premier league in Ghana. Future researchers are encouraged to re-validate the stress appraisal measure with a different sample to contribute to the understanding of the applicability of the instrument in non-western populations.

Is Self-Transcendence Philanthropic? Graded Response Model Approach.

This study reveals that strong feelings of altruism were found to be statistically significant in explaining prosocial and pro-environmental behaviors. However, this was not the case for the latent trait biosphere in explaining pro-environmental behavior (e.g., past volunteering in clean-up activities). Regardless of whether they are overseas graduates or not, subjects in this study are more altruistic than biospheric by nature. Using the Graded Response Model (GRM) approach, the study found that the biosphere and altruism are obviously independent of each other and merging them into one dimension, in this instance referred to as "self-transcendence," makes the construct less reliable. That is why this study in consistence with previous studies could not detect the effect of self-transcendence statistically, as it affects both the past volunteering in environmental affairs and the past volunteering in social welfare.